Parameter-dependent model-blending with multi-expert based machine learning and proxy sites

ABSTRACT

A parameter-based multi-model blending method and system are described. The method includes selecting a parameter of interest among parameters estimated by each of a set of individual models, running the set of individual models with a range of inputs to obtain a range of estimates of the parameters from each of the set of individual models, and identifying, for each of the set of individual models, critical parameters among the parameters estimated, the critical parameters exhibiting a specified correlation with an error in estimation of the parameter of interest. For each subspace of combinations of the critical parameters, obtaining a parameter-based blended model is based on blending the set of individual models in accordance with the subspace of the critical parameters, the subspace defining a sub-range for each of the critical parameters.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under DE-EE0006017awarded by Department of Energy. The Government has certain rights tothis invention.

BACKGROUND

The present invention relates to model blending, and more specifically,to parameter-dependent model-blending with multi-expert based machinelearning and proxy sites.

Physical models that are based on principles of physics and chemistryand which are used to forecast parameters or conditions in a widevariety of arenas are known. Meteorological models may be used toforecast weather, for example. These models may include input parameterssuch as pressure, temperature, and wind velocity and provide estimatesor predictions of output parameters. Corrosion models may forecastpipeline corrosion, as another example. These models may include inputparameters such as temperature, gas concentrations, pressure, and flowconditions. Different physical models that provide the same predictedoutput condition or parameter may be blended to improve the predictionoffered by any one of the models individually.

SUMMARY

According to an embodiment, a method of performing parameter-basedmulti-model blending includes selecting a parameter of interest amongparameters estimated by each of a set of individual models; running,using a processor, the set of individual models with a range of inputsto obtain a range of estimates of the parameters from each of the set ofindividual models; identifying, for each of the set of individualmodels, critical parameters among the parameters estimated, the criticalparameters exhibiting a specified correlation with an error inestimation of the parameter of interest; obtaining, for each subspace ofcombinations of the critical parameters, a parameter-based blended modelbased on blending the set of individual models in accordance with thesubspace of the critical parameters, the subspace defining a sub-rangefor each of the critical parameters.

According to another embodiment, a system to perform parameter-basedmulti-model blending includes an input interface configured to receiveinputs, the inputs including the parameter of interest among parametersestimated by each of a set of individual models; and a processorconfigured to run the set of individual models with a range of inputs toobtain a range of estimates of the parameters from each of the set ofindividual models, identify, for each of the set of individual models,critical parameters among the parameters estimated, the criticalparameters exhibiting a specified correlation with an error inestimation of the parameter of interest, and obtain, for each subspaceof combinations of the critical parameters, a parameter-based blendedmodel based on blending the set of individual models in accordance withthe subspace of the critical parameters, the subspace defining asub-range for each of the critical parameters.

According to yet another embodiment, a non-transitory computer programproduct has computer readable instructions stored thereon which, whenexecuted by a processor, cause the processor to implement a method ofmulti-model blending. The method includes selecting a parameter ofinterest among parameters estimated by each of a set of individualmodels; running the set of individual models with a range of inputs toobtain a range of estimates of the parameters from each of the set ofindividual models; identifying, for each of the set of individualmodels, critical parameters among the parameters estimated, the criticalparameters exhibiting a specified correlation with an error inestimation of the parameter of interest; and obtaining, for eachsubspace of combinations of the critical parameters, a parameter-basedblended model based on blending the set of individual models inaccordance with the subspace of the critical parameters, the subspacedefining a sub-range for each of the critical parameters.

According to yet another embodiment, a method of classifying sites toobtain a proxy site for a site of interest includes determining criticalparameters among parameters estimated by one or more models associatedwith each of the sites, the parameters including the criticalparameters, a parameter of interest, and other parameters, and thecritical parameters being determined to be critical in estimation of theparameter of interest according to the one or more models; grouping twoor more sites together in a same group when the two or more sites havesame critical parameters; for each group of the two or more sites,classifying the two or more sites by type; correlating the typeassociated with each site with latitude, longitude, and elevation of thesite; and obtaining, using a processor, the proxy site for the site ofinterest by determining the type correlated with latitude, longitude,and elevation of the site of interest.

According to yet another embodiment, a system to classify sites toobtain a proxy site for a site of interest includes an input interfaceconfigured to receive an input of a number of types; and a processorconfigured to determine critical parameters among parameters estimatedby one or more models associated with each of the sites, the parametersincluding the critical parameters, a parameter of interest, and otherparameters, and the critical parameters being determined to be criticalin estimation of the parameter of interest according to the one or moremodels, to group two or more sites together in a same group when the twoor more sites have same critical parameters, to classify the two or moresites of each group into a type, the number of types being specified inthe input, to correlate the type associated with each site withlatitude, longitude, and elevation of the site, and to obtain a proxysite for the site of interest by determining the type correlated withlongitude, latitude, and elevation of the site of interest.

According to yet another embodiment, a non-transitory computer programproduct has computer readable instructions stored thereon which, whenexecuted by a processor, cause the processor to implement a method ofclassifying sites to obtain a proxy site for a site of interest. Themethod includes determining critical parameters among parametersestimated by one or more models associated with each of the sites, theparameters including the critical parameters, a parameter of interest,and other parameters, and the critical parameters being determined to becritical in estimation of the parameter of interest according to the oneor more models; grouping two or more sites together in a same group whenthe two or more sites have same critical parameters; for each group ofthe two or more sites, classifying the two or more sites by type;correlating the type associated with each site with latitude, longitude,and elevation of the site; and obtaining the proxy site for the site ofinterest by determining the type correlated with latitude, longitude,and elevation of the site of interest.

According to yet another embodiment, a method to determine a blendedforecasting model includes storing historical data, the historical dataincluding estimates and measurements of a parameter of interest andestimates of critical parameters, the critical parameters determined tobe critical to an estimate of the parameter of interest; training aplurality of machine learning models with respective machine learningalgorithms using training data that includes a first set of parametervalues associated with a first range of time points, the first set ofparameter values being obtained from the historical data; obtainingestimates of the parameter of interest with each of the machine learningmodels using a second set of parameter values associated with a secondrange of time points, the second set of parameter values being obtainedfrom the historical data; determining, using a processor, a mostaccurate machine learning model among the machine learning models ateach time point in the second range of time points; and determining theblended forecasting model based on the most accurate machine learningmodel determined for each time point in the second range of time points.

According to yet another embodiment, a multi-expert based machinelearning system to determine a blended forecasting model includes amemory device to store historical data of parameters, the historicaldata including estimates and measurements of a parameter of interest andestimates of critical parameters determined to be critical to anestimate of the parameter of interest; and a processor configured totrain a plurality of learning models with respective machine learningalgorithms using training data that includes a first set of parametervalues associated with a first range of time points obtained from thehistorical data, to obtain estimates of the parameter of interest witheach of the machine learning models using a second set of parametervalues associated with a second range of time points, the second set ofparameter values being obtained from the historical data, to determine amost accurate machine learning model among the machine learning modelsat each time point in the second range of time points, and to determinethe blended forecasting model from the most accurate machine learningmodels.

According to yet another embodiment, a non-transitory computer programproduct has computer readable instructions stored thereon which, whenexecuted by a processor, cause the processor to implement a method ofdetermining a blended forecasting model. The method includes obtaininghistorical data, the historical data including estimates andmeasurements of a parameter of interest and estimates of criticalparameters, the critical parameters determined to be critical to anestimate of the parameter of interest; training a plurality of machinelearning models with respective machine learning algorithms usingtraining data that includes a first set of parameter values associatedwith a first range of time points, the first set of parameter valuesbeing obtained from the historical data; obtaining estimates of theparameter of interest with each of the machine learning models using asecond set of parameter associated with a second range of time points,the second set of parameter values being obtained from the historicaldata; determining a most accurate machine learning model among themachine learning models at each time point in the second range of timepoints; and determining the blended forecasting model based on the mostaccurate machine learning model determined for each time point in thesecond range of time points.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention. For a better understanding of the invention with theadvantages and the features, refer to the description and to thedrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter which is regarded as the invention is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features, and advantages ofthe invention are apparent from the following detailed description takenin conjunction with the accompanying drawings in which:

FIG. 1 is a process flow of a method of performing parameter-based modelblending according to embodiments;

FIGS. 2-5 illustrate the information and processes used to identifycritical parameters according embodiments, in which:

FIG. 2 illustrates exemplary results of one individual model to beblended according to embodiments;

FIG. 3 illustrates an exemplary visualization of parameters thatfacilitates comparison of the correlation of each parameter with theerror in estimating the parameter of interest;

FIG. 4 illustrates an exemplary visualization of parameters thatfacilitates comparison of the correlation of a pair of parameters withthe error in the estimate of the parameter of interest; and

FIG. 5 shows inter-model second order error dependence;

FIG. 6 is a process flow of a multi-expert based machine learningtechnique according to an embodiment;

FIG. 7 shows a process flow of a method of classifying sites andobtaining proxy sites according to an embodiment; and

FIG. 8 is a block diagram of a multi-model blending system according toan embodiment of the invention

DETAILED DESCRIPTION

As noted above, a model may be used to forecast or estimate values ofparameters or future conditions. Further, outputs of more than one modelmay be blended to improve the prediction provided by individual models,because no individual model is likely to be accurate in all situations.Embodiments of the systems and methods detailed herein relate to aparameter-dependent blend of individual models. For a given subspace ofparameter values (a given situation), the performance of each individualmodel with respect to the parameter of interest is used to determine theappropriate blend of models for the given situation. As such,parameter-based (or situation-specific) errors may be essentiallyeliminated in the blended model. According to embodiments, amulti-expert based machine learning is used to obtain the blended modelfor each situation. According to embodiments, proxy sites may beidentified for purposes of obtaining training data sets. When trainingthe blended model, if historical data is not available at the site ofinterest, data from one or more proxy sites may be used instead.

FIG. 1 is a process flow of a method of performing parameter-based modelblending according to embodiments detailed herein. At block 110,selecting the estimate of interest and the site are performed. Forpurposes of explanation, a specific example of the estimate of interestbeing global horizontal irradiance (GHI) is discussed. The exemplarymodels discussed herein that estimate or predict GHI have differentinputs based on the site of interest (physical location). However, othermodels may not be site-specific. As noted above, the discussion hereinapplies to any number of types of models and any estimate a parameter ofinterest associated with those models. Running individual models withdifferent input values, at block 120, results in a range of predictionsor estimates of parameters by each model. To be clear, while onlyestimate may be used herein, the models (individual and blended) mayprovide predictions of future parameter values as well as estimates ofparameter values corresponding with a time at which input values wereobtained. The range of estimates of parameters includes the estimate ofthe parameter of interest (a range of estimates of the parameter ofinterest). As detailed further below, identifying critical parameters,at block 130, includes identifying, among the parameters estimated bythe individual models, those parameters that have the greatest influenceon the error in the estimate of the parameter of interest. The parameterof interest itself may be one of the critical parameters. Once thecritical parameters are identified, setting a subspace of the criticalparameters, at block 140, is done iteratively and includes considering acombination of a sub-range of each critical parameter per iteration. Thesub-range of values considered for a given critical parameter need notbe continuous. As further discussed below, dependence of the error inthe estimation of the parameter of interest may be similar for differentsets of values of a critical parameter. Obtaining the parameter-basedblended model, at block 160, may involve obtaining a training data setat block 150 for use with machine learning. This machine learning,according to embodiments detailed with reference to FIG. 6 below, mayinclude multi-expert based machine learning. As further detailed belowwith reference to FIG. 7, the training data set may be from a proxy siterather than from the site selected at block 110.

FIGS. 2-5 illustrate the information and processes used to identifycritical parameters according embodiments. FIGS. 2-5 relate to blocks120 and 130 of FIG. 1.

FIG. 2 illustrates exemplary results of one individual model that willbe blended according to embodiments detailed herein. Four differentgraphs 210, 220, 230, 240 are shown. Each graph shows the first ordererror dependence in GHI estimation corresponding with a range ofestimations for another parameter. These graphs 210-240 result fromfunctional analysis-of-variance (FANOVA) in the first order. FANOVA is aknown technique of using statistical models to analyze variance andexplain observations. Its application according to the presentembodiment is to build a statistical model of prediction error (inpredicting the parameter of interest by a given individual model) as afunction of all input parameters. Error in GHI estimate may be computedas:E=F(x ₁ , x ₂ , . . . , x _(n))  [EQ. 1]EQ. 1 provides the model forecast error (E) of the parameter of interest(GHI in this example). x₁, x₂, . . . , x_(n) are the other n parametersthat are also forecast (predicted or estimated) by the individual model.The statistical models are generally too noisy to be used directly andare therefore decomposed to 0^(th), 1^(st), 2^(nd), and higher orderdependence of forecast error as follows:

$\begin{matrix}{F = {f_{0} + {\sum\limits_{i}\;{f_{i}\left( x_{i} \right)}} + {\sum\limits_{i \neq j}\;{f_{i,j}\left( {x_{i},x_{j}} \right)}} + \ldots}} & \left\lbrack {{EQ}.\mspace{14mu} 2} \right\rbrack\end{matrix}$The first order dependence (of error in estimating the parameter ofinterest) on a single variable (another parameter estimated by the sameindividual model) is then given by:f _(i) =∫F(x ₁ , . . . , x _(n))dx ₁ . . . dx _(i−1) d _(i+1) dx _(n) −f₀  [EQ. 3]

Graphs 210-240 indicate this first order dependence f_(i) for fourdifferent parameters (values of i) in FIG. 2. Graph 210 shows the firstorder error dependence in GHI estimates corresponding with estimates ofground pressure. Graph 210 indicates that the error decreases as theestimated ground pressure increases. Graph 220 shows the first ordererror dependence of GHI estimates corresponding with estimates of columncloud liquid water. Graph 220 indicates that error in the estimate ofGHI decreases sharply when the estimated column cloud liquid water isover 0.5 kilograms per meters squared (kg/m²). Graph 230 shows the firstorder error dependence of GHI estimates corresponding with estimates ofcloud bottom height, and graph 240 shows the first order errordependence of GHI estimates corresponding with estimates of cloud topheight. Graph 230 indicates a sharp rise in the error of the GHIestimate above a cloud bottom height estimate of 9000 meters (m), andgraph 240 indicates an increase in the GHI estimate error above a cloudtop height estimate of 10000 m. The error in the GHI estimate is a firstorder error because it depends on only one parameter, and the effects ofother parameters on the estimation error are averaged out in EQ. 3.

Graph 230 is an example of sub-ranges of parameter values that havesimilar correlations with estimation error of the parameter of interest.That is, cloud bottom height values below 2000 have a similarcorrelation with first order GHI estimate error as do cloud bottomheight values between 8,500 and 10,000 m. Thus, if cloud bottom heightwere determined to be a critical parameter, a training data set,discussed further below, would involve a sub-range that includes both 0to 2,000 m and 8,500 to 10,000 m. This is an example of a non-continuoussub-range.

FIG. 3 illustrates an exemplary visualization of parameters thatfacilitates comparison of the correlation of each parameter with thefirst order error in estimating the parameter of interest (GHI in theexample). Each bar shown in FIG. 3 indicates the standard deviation ofthe first order error in GHI estimate corresponding with a givenparameter. That is, for graph 210 (FIG. 2), for example, the mean valueof first order GHI estimate error is determined, and the deviation ofeach point on the graph 210 from the mean value is used to computestandard deviation. This computed standard deviation (obtained fromgraph 210) is shown as the bar associated with parameter 9 (groundpressure) in FIG. 3. Thus, the standard deviation is a measure of thespread in GHI estimate error dependence corresponding to each parameterand is given by:

$\begin{matrix}{{standard\_ deviation} = \sqrt{\frac{\sum\limits_{i = 1}^{N}\;\left( {X_{i} - {mean}} \right)^{2}}{N - 1}}} & \left\lbrack {{EQ}.\mspace{14mu} 4} \right\rbrack\end{matrix}$In EQ. 4, N is the total number of first order error dependence valuesassociated with a given parameter (e.g., number of points in graph 210,FIG. 2), and X_(i) refers to each first order error dependence value(each value in graph 210). All of the parameters (numbered 1 through 24)are estimated by an individual model at the site of interest. FIG. 3indicates that ground pressure (parameter 9) and zenith angle (parameter23) are important parameters in terms of first order error in GHIestimation. This identification of influential parameters may be basedon setting a threshold for the standard deviations of the GHI errordependence on different parameters, for example. To identify criticalparameters, the GHI second order error dependence on parameters isexamined, as well, as described below.

FIG. 4 illustrates an exemplary visualization of the standard deviationof the second order error in the estimate of the parameter of interest(e.g., GHI). FIG. 5 shows exemplary second order error dependence. Asdiscussed with regard to FIGS. 2 and 3, FIG. 4 represents the result ofdetermining the mean value of second order estimate error and then thestandard deviation based on the deviation from that mean value at eachpoint. While the standard deviation of the first order GHI estimationerror dependence is based on one parameter (as discussed with referenceto FIG. 3), the standard deviation of the second order GHI estimationerror dependence (W/m²) is based on a combination of two parameters, asindicated by each bar in FIG. 4. Thus, the standard deviation of thesecond order GHI error correlated with a combination of parameter 7(wind speed) and parameter 9 (ground pressure) is shown as 410. Thestandard deviation of the second order GHI error correlated withparameter 9 (ground pressure) and parameter 1 (GHI) is shown as 420. Bargraphs 430 and 440 show the standard deviation of the second order GHIerror correlated with a combination of parameter 20 (clear sky GHI) andparameter 1 (GHI) and with a combination of parameter 23 (zenith angle)and parameter 1 (GHI), respectively. A threshold value may be used toselect these four combinations as influential combinations of parameterswith respect to estimation error for GHI. The FANOVA second orderdependence (derived from EQ. 2) is given by:f _(i,j) =∫F(x ₁ , . . . , x _(n))dx ₁ . . . dx _(i−1) dx _(i+1) . . .dx _(j−1) dx _(j+1) . . . dx _(n) −f _(i)(x _(i))−f _(j)(x _(j))−f₀  [EQ. 5]As noted above, the information obtained from FIGS. 2-4 illustrate firstand second order GHI estimation error associated with one individualmodel, and the process of examining the parameters is repeated for otherindividual models. The process of examining the parameters may also beextend to for higher order (third order or above) error dependences. Inaddition, cross-model parameter dependence may also be considered.

FIG. 5 shows inter-model second order error dependence. Diffusehorizontal irradiance (DHI) estimated by one individual model (model1)and GHI estimated by another model (model2) are shown with second orderGHI estimate error indicated by the grayscale. According to FIG. 5, whenDHI estimated by model 1 is high and GHI estimated by model 2 is high,the estimates are in the region indicated by 510, which is correlatedwith high second order error in GHI estimation. On the other hand, inthe middle region indicated by 520, in which both the DHI estimate bymodel 1 and the GHI estimate by model 2 are approximately in the middleof the range of estimated values, the second order GHI estimation errorcorrelated with this region (520) is the lowest. FIG. 5 illustrates astatistical correlation between these two independent models.

Based on the first and second order errors and on inter-model errorcorrelation examined as exemplified in the discussion above, criticalparameters are identified. These critical parameters are determined tohave the highest (e.g., above a threshold) correlation with the error inestimating the parameter of interest. The same parameters may not becritical parameters in each individual model. However, the processesdiscussed above identify parameters that are deemed critical in at leastone individual model. If the number of these critical parameters is onlyone or two, then blending the individual models may be achieved in astraight-forward manner by a weighted linear combination, for example.In most situations, based on the number of critical parameters, theblended model is obtained through machine learning with trainingdatasets. The training data sets consider available historical datawhich fall in a number of subspaces, where each subspace is a particularcombination of the critical parameters, each critical parameter set at aparticular sub-range of its values. As noted above, a sub-range is notnecessarily a continuous range of values. An exemplary embodiment fordividing the total historical data into subspaces involves using theestimation error of the parameter of interest. That is, within asubspace, the estimation error of the parameter of interest is similar.Once trained, the resulting blended model may be applied for estimationwhere the critical parameters fall in the same subspace. According toembodiments detailed below, the machine learning may be accomplished bya multi-expert based machine learning system. Additionally, according toembodiments detailed below, the issue of obtaining training datasets isaddressed. That is, when (historical) training data is not available forthe site of interest, proxy sites that provide comparable and sufficienttraining data to be used in generating a blended model that may then beapplied to the site of interest are needed.

FIG. 6 is a process flow of a multi-expert based machine learningtechnique according to an embodiment. The multi-expert based machinelearning technique determines the most appropriate machine learningalgorithm for a given situation (for a given subspace or range of valuesof the critical parameters). As detailed below, the multi-expert basedmachine learning determines the best machine learning algorithm withwhich to train a machine learning model for each situation. Initially,all the candidate machine leaning algorithms are used to train therespective different machine learning models 620 a through 620 z usingpart of the available historical data 610 (estimates of all parameters(including the parameter of interest 612 and critical parameters 615)and, additionally, measurements of the parameter of interest 617). Onlypart of the available historical data 610 is used so that the remaininghistorical data 610 may be used to test the machine learning models 620.For example, if a year's worth of historical data 610 is available, onlythe first eleven months of data may be used to train the machinelearning models 620. Exemplary machine learning algorithms 620 include alinear regression, random forest regression, gradient boostingregression tree, support vector machine, and neural networks. Theestimates or predictions 630 a through 630 z of the parameter ofinterest (at various points of time) by each machine learning model 620a through 620 z, respectively, are obtained for the period of time forwhich historical data 610 is available but was not used for training(e.g., the remaining month of the year in the example noted above). Ateach point in time, the machine learning model and correspondingcritical parameters 620/615 associated with the most accurate prediction630 among all the predictions 630 is determined. The accuracy isdetermined based on a comparison of the estimates 630 a through 630 zwith the historical data 610 available for the period during which theestimates 630 a through 630 z are obtained. The resulting set of (mostaccurate) machine learning model and critical parameters 620/615combinations is stored as the combinations 640 and is used to obtain theparameter-based blended model. That is, when the blended model is to beused, all critical parameters are estimated by all individual models.Based on the estimated ranges for the critical parameters 615, thecorresponding machine learning model 620 from the stored combinations640 is selected for use.

In alternate embodiments, the critical parameters 615 may be used toobtain the parameter-based blended model using another machine learningtechnique. That is, the combinations (640) of machine learning model andcritical parameters 620/615 may be used to train a classificationmachine learning model to correlate the machine learning model 620 withcritical parameters 615. Once the classification machine learning modelis trained, inputting critical parameters 615 will result in obtainingthe appropriate machine learning model 620 (parameter-based blendedmodel).

In yet another embodiment, a single machine learning model 620 may beselected from among the set of most accurate machine learning models620. For example, the machine learning model 620 that is most often themost accurate machine learning model 620 (for more points in time) maybe selected as the parameter-based blended model. According to thisembodiment, no correlation of machine learning model 620 to criticalparameters 615 is needed.

The training data 610 discussed with reference to FIG. 6 may behistorical and measured data from the site of interest. However, in somesituations, training data specific to the site of interest (selected atblock 110, FIG. 1) may not be available. The lack of site-specifictraining data may be addressed in a number of ways. According to anembodiment detailed below, sites are analyzed for similarities andcategorized such that proxy sites may be identified when sites ofinterest fail to have training data.

FIG. 7 shows a process flow of a method of classifying sites andobtaining proxy sites according to an embodiment. At block 710,determining critical parameters for each site with availablemeasurements may include performing the processes discussed above withreference to FIGS. 2-5. Grouping sites together that have the samecritical parameters is performed at block 720. The sites within a givengroup must have all critical parameters in common rather than just asubset. For each group of sites, a further classification is thenperformed at block 730 that involves classifying the sites by type. Thisclassification may be based on the estimation error dependence (of theparameter of interest) on the corresponding critical parameters of thegroup of sites, as detailed below. In alternate embodiments, staticinformation of the site such as land type, elevation, latitude,longitude may be used in addition to the estimation error dependence forsite classification (as additional coefficients). This classification atblock 730 sorts the sites by type. At block 740, correlating the type ofa site with its latitude, longitude, and elevation may include traininga supervised classification model that correlates site type withlatitude, longitude, and elevation. Exemplary algorithms for trainingthe supervised classification model include the random forest algorithm,regression tree, support vector machine, and neural networks. Thetraining data used to train the classification model is each site typedetermined at block 730 (response variable) and corresponding latitude,longitude, and elevation (predictor variables). A given site type mayhave multiple sites and, thus, multiple associated sets of latitude,longitude, and elevation combinations. Once the classification model istrained at block 740, determining a site type, at block 750, of any siteis a matter of entering the latitude, longitude, and elevation of thatsite to the classification model for output of the site type. By usingthe site type, proxy sites (sites of the same type) may be identifiedfrom the original set of sites for which measurements were available (atblock 710). As noted above with reference to FIG. 1, block 150, trainingdata may be obtained from a proxy site when the site of interest has nohistorical or measured data available. One or more proxy sites may beused to provide the training data.

The classification at block 730 may begin with the first and secondorder error (in the estimate of the parameter of interest) dependencedetermined using FANOVA as discussed with reference to embodimentsabove. Polynomial models are fit to the first and second order errordependence for each site. For example, a linear model is fit to thefirst order error estimate and a quadratic model is fit to the secondorder error estimate. Thus, a first order error dependence curve (e.g.,graph 210 in FIG. 2) is translated into two polynomial coefficients (theslope and intercept of the line fit to the graph) and a second ordererror dependence surface is translated to six coefficients. Accordingly,an individual site is associated with a set of polynomial coefficientscorresponding to all of its first and second order error dependences ofthe parameter of interest. Using an unsupervised clustering machinelearning algorithm (e.g., method of moments, k-means clustering,Gaussian mixture model, neural network), each site may be classifiedaccording to its set of coefficients. An input to the clustering machinelearning algorithm is the number of total types of sites into which tosort the available sites. Given this number, the clustering algorithmmay compute and use a measure of similarity among sets of coefficients(each set associated with a different site) to sort the sites.

In an alternative embodiment, the classification at block 730 and,specifically, the generation of the coefficients may be donedifferently. For each site, a linear model of the parameter of interest(y) may be fit to all or a subset of the critical parameters (x₁ throughx_(n)) associated with the site. The coefficients (a₁ through a_(n)) maythen be determined from the linear model (y=a₁x₁+a₂x₂+ . . .+a_(n)x_(n)). This set of coefficients (a₁ . . . a_(n)) rather than thecoefficients obtained from the first order error dependence curve andsecond order error dependence surface, as discussed above, may be usedwith the clustering machine learning algorithm to sort the sites intosites types.

FIG. 8 is an overview of a multi-model blending system 800 according toan embodiment of the invention. The system 800 includes an inputinterface 813, one or more processors 815, one or more memory devices817, and an output interface 819. The system 800 may communicate,wirelessly, through the internet, or within a network, for example, withone or more devices 820A through 820N (generally, 820). The otherdevices 820 may be other systems 800 or sources of training data ormodel outputs. That is, not all of the models may be executed within themulti-model blending system 800. Instead, one or more individual modelsmay be implemented by another device 820 and the output (predicted orestimated parameters) provided to the input interface 813. For example,in the exemplary case of multi-model blending of meteorological models,device A 820A may be The National Weather Service or anotherorganization that executes a model to forecast weather and provides themodel output. The processes detailed above (including identifyingcritical parameters and classifying site types) may be executed by thesystem 800 alone or in combination with other systems and devices 820.For example, the input interface 813 may receive information about theparameter of interest and the site of interest (and the number of sitetypes), as well as receive training data or model outputs. The processormay determine the critical parameters for a set of models providing agiven parameter of interest, as detailed above.

All of the embodiments discussed herein ultimately improve the area inwhich the forecast or estimate is provided. For example, when theindividual models used, as described above, relate to weatherforecasting, the embodiments detailed herein improve the weatherforecast, or when the individual models relate to corrosion forecasting,the embodiments detailed herein improve the forecast and, thus,reliability in the pipeline industry.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of onemore other features, integers, steps, operations, element components,and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated

The flow diagrams depicted herein are just one example. There may bemany variations to this diagram or the steps (or operations) describedtherein without departing from the spirit of the invention. Forinstance, the steps may be performed in a differing order or steps maybe added, deleted or modified. All of these variations are considered apart of the claimed invention.

While the preferred embodiment to the invention had been described, itwill be understood that those skilled in the art, both now and in thefuture, may make various improvements and enhancements which fall withinthe scope of the claims which follow. These claims should be construedto maintain the proper protection for the invention first described.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method of performing parameter-basedmulti-model blending, the method comprising: selecting a parameter ofinterest among parameters estimated by each of a set of individualmodels; running, using a processor, the set of individual models with arange of inputs to obtain a range of estimates of the parameters fromeach of the set of individual models, wherein each individual model fromthe set of individual models is associated with a particular site of aset of sites and wherein each site from the set of sites has acorresponding latitude, longitude, and elevation; identifying, for eachof the set of individual models, critical parameters among theparameters estimated, the critical parameters exhibiting a specifiedcorrelation with an error in estimation of the parameter of interest,wherein the identifying the critical parameters includes examining firstorder dependence and second order dependence of the error in theestimation of the parameter of interest associated with each of theparameters estimated by each of the set of individual models and whereinexamining the first order dependence includes translating a first ordererror dependence curve into two coefficients and examining the secondorder dependence includes translating a second order error dependencesurface into six coefficients; correlating a type associated with eachsite with the latitude, the longitude, and the elevation of the site byobtaining, for each subspace defined as a particular combination ofsub-ranges of a range of values of the critical parameters, aparameter-based blended model by training a classification machinelearning model to correlate one or more machine learning models withcorresponding values derived from the subspace of the criticalparameters, the subspace defining a sub-range for each of the criticalparameters, and inputting the corresponding values derived from thesubspace of the critical parameters into the trained classificationmachine learning model, wherein the correlating includes training theclassification machine learning model to correlate the site type withthe latitude, the longitude, and the elevation of the site; forecastingor estimating the parameter of interest using the parameter-basedblended model selected, by the trained classification machine learningmodel, from among the one or more machine learning models based on thecritical parameters; and selecting a site of interest from the set ofsites such that the range of inputs correspond with the site ofinterest.
 2. The method according to claim 1, wherein the identifyingthe critical parameters includes calculating a variance from the firstorder dependence associated with each of the parameters estimated byeach of the set of individual models.
 3. The method according to claim2, wherein the identifying the critical parameters includes identifyingparameters among the parameters estimated by each of the set ofindividual models with the associated variance exceeding a thresholdvalue.
 4. The method according to claim 1, wherein the identifying thecritical parameters also includes examining higher order dependence ofthe error in the estimation of the parameter of interest associated withcombinations of parameters estimated by each of the set of individualmodels.
 5. The method according to claim 1, wherein the identifying thecritical parameters also includes examining inter-model second orderdependence of the error in the estimation of the parameter of interestassociated, the inter-model second order dependent of the errorreferring to a correlation between error dependence in estimation of theparameter of interest by a first model among the set of individualmodels on a first parameter and error dependence in estimation of theparameter of interest by a second model among the set of individualmodels on a second parameter.
 6. The method according to claim 1,wherein the obtaining the parameter-based blended model for eachsubspace of combinations of the critical parameters includesdetermining, by the trained classification machine learning model, amost accurate machine learning model among the one or more machinelearning models for each combination of critical parameters.
 7. A systemto perform parameter-based multi-model blending, the system comprising:a memory device configured to store inputs, the inputs including theparameter of interest among parameters estimated by each of a set ofindividual models; and a processor configured to run the set ofindividual models with a range of inputs to obtain a range of estimatesof the parameters from each of the set of individual models, whereineach individual model from the set of individual models is associatedwith a particular site of a set of sites and wherein each site from theset of sites has a corresponding latitude, longitude, and elevation,identify, for each of the set of individual models, critical parametersamong the parameters estimated, the critical parameters exhibiting aspecified correlation with an error in estimation of the parameter ofinterest, wherein the processor identifies the critical parameters basedon examining first order dependence and second order dependence of theerror in the estimation of the parameter of interest associated witheach of the parameters estimated by each of the set of individual modelsand wherein he first order dependence is examined based on translating afirst order error dependence curve into two coefficients and wherein thesecond order dependence is examined based on translating a second ordererror dependence surface into six coefficients, correlate a typeassociated with each site with the latitude, the longitude, and theelevation of the site by obtaining, for each subspace defined as aparticular combination of sub-ranges of a range of values of thecritical parameters, a parameter-based blended model by training aclassification machine learning model to correlate one or more machinelearning models with corresponding values derived from the subspace ofthe critical parameters, the subspace defining a sub-range for each ofthe critical parameters, and inputting the corresponding values derivedfrom the subspace of the critical parameters into the trainedclassification machine learning model, wherein the correlating includestraining the classification machine learning model to correlate the sitetype with the latitude, the longitude, and the elevation of the site,forecast or estimate the parameter of interest using the parameter-basedblended model selected, by the trained classification machine learningmodel, from among the one or more machine learning models based on thecritical parameters; and select a site of interest from the set of sitessuch that the range of inputs correspond with the site of interest. 8.The system according to claim 7, wherein the processor identifies thecritical parameters based on calculating a variance from the first orderdependence associated with each of the parameters estimated by each ofthe set of individual models.
 9. The system according to claim 8,wherein the processor identifies the critical parameters based onidentifying parameters among the parameters estimated by each of the setof individual models with the associated variance exceeding a thresholdvalue.
 10. The system according to claim 7, wherein the processoridentifies the critical parameters based additionally on examininghigher order dependence of the error in the estimation of the parameterof interest associated with combinations of parameters estimated by eachof the set of individual models.
 11. The system according to claim 7,wherein the processor identifies the critical parameters basedadditionally on examining inter-model second order dependence of theerror in the estimation of the parameter of interest associated, theinter-model second order dependent of the error referring to acorrelation between error dependence in estimation of the parameter ofinterest by a first model among the set of individual models on a firstparameter and error dependence in estimation of the parameter ofinterest by a second model among the set of individual models on asecond parameter.
 12. The system according to claim 7, wherein theprocessor obtains the parameter-based blended model by determining, bythe trained classification machine learning model, a most accuratemachine learning model among the one or more machine learning models foreach combination of critical parameters.
 13. A non-transitory computerprogram product having computer readable instructions stored thereonwhich, when executed by a processor, cause the processor to implement amethod of multi-model blending, the method comprising: selecting aparameter of interest among parameters estimated by each of a set ofindividual models; running the set of individual models with a range ofinputs to obtain a range of estimates of the parameters from each of theset of individual models, wherein each individual model from the set ofindividual models is associated with a particular site of the set ofsites and wherein each site from the set of sites has a correspondinglatitude, longitude, and elevation; identifying, for each of the set ofindividual models, critical parameters among the parameters estimated,the critical parameters exhibiting a specified correlation with an errorin estimation of the parameter of interest, wherein the identifying thecritical parameters includes examining first order dependence and secondorder dependence of the error in the estimation of the parameter ofinterest associated with each of the parameters estimated by each of theset of individual models and wherein examining the first orderdependence includes translating a first order error dependence curveinto two coefficients and examining the second order dependence includestranslating a second order error dependence surface into sixcoefficients; correlating a type associated with each site with thelatitude, the longitude, and the elevation of the site by obtaining, foreach subspace defined as a particular combination of sub-ranges of arange of values of the critical parameters, a parameter-based blendedmodel by training a classification machine learning model to correlateone or more machine learning models with corresponding values derivedfrom the subspace of the critical parameters, the subspace defining asub-range for each of the critical parameters, and inputting thecorresponding values derived from the subspace of the criticalparameters into the trained classification machine learning model,wherein the correlating includes training the classification machinelearning model to correlate the site type with the latitude, thelongitude, and the elevation of the site; forecasting or estimating theparameter of interest using the parameter-based blended model selected,by the trained classification machine learning model, from among the oneor more machine learning models based on the critical parameters; andselecting a site of interest from the set of sites such that the rangeof inputs correspond with the site of interest.
 14. The non-transitorycomputer program product according to claim 13, wherein the identifyingthe critical parameters also includes examining higher order dependenceof the error in the estimation of the parameter of interest associatedwith combinations of parameters estimated by each of the set ofindividual models.
 15. The non-transitory computer program productaccording to claim 13, wherein the identifying the critical parametersalso includes examining inter-model second order dependence of the errorin the estimation of the parameter of interest associated, theinter-model second order dependent of the error referring to acorrelation between error dependence in estimation of the parameter ofinterest by a first model among the set of individual models on a firstparameter and error dependence in estimation of the parameter ofinterest by a second model among the set of individual models on asecond parameter.
 16. The non-transitory computer program productaccording to claim 13, wherein the obtaining the parameter-based blendedmodel for each subspace of combinations of the critical parametersincludes determining, by the trained classification machine learningmodel, a most accurate machine learning model among the one or moremachine learning models for each combination of critical parameters.